Repository navigation
Conversation
This was referenced Oct 4, 2026
zzylol
force-pushed
the
stack/509-y2-filtered-aggregates
branch
from
October 5, 2026 06:21
4c68af1 to
e1a4e6d
Compare
zzylol
force-pushed
the
stack/509-y2b-filtered-pass1
branch
2 times, most recently
from
October 8, 2026 16:47
a90e186 to
66bf263
Compare
zzylol
force-pushed
the
stack/509-y2-filtered-aggregates
branch
from
October 10, 2026 14:32
14ef881 to
6ba89f0
Compare
zzylol
force-pushed
the
stack/509-y2b-filtered-pass1
branch
from
October 10, 2026 14:32
66bf263 to
2faa0de
Compare
3 tasks
Pass 1 listed alternatives for a filtered single-measure aggregate (SQL FILTER, or the IS NOT NULL a COUNT(nullable) implies) but every non-pass-through choice failed to compose. The measure's filter now becomes SummaryAgg.filter, so exact accumulators and sketches (Hydra included) compose, build, and execute. Coverage stays whole-source. A FinalizeExactAccumulator or quantile SummaryEstimate over a filtered build is declared nullable, as the filtered aggregate it realizes is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…loads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
Pass 1 now builds a filtered SummaryAgg, so the #573 §4.2.2 worked example runs whole: the FILTER range and the renamed value set move to the selection, and the expression condition stays in the definition. The quantile state is read from the Stage 1 logical candidates, which build it whatever plan the cost model selects. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol
force-pushed
the
stack/509-y2-filtered-aggregates
branch
from
October 11, 2026 17:11
6ba89f0 to
579b2d4
Compare
zzylol
force-pushed
the
stack/509-y2b-filtered-pass1
branch
from
October 11, 2026 17:11
2faa0de to
0bbfc1b
Compare
zzylol
added a commit
that referenced
this pull request
Oct 13, 2026
… targets Stage 1 composition realizes every #561 Pass 1 choice. A measure filter becomes SummaryAgg.filter (the realization rule #608 carried, with its filtered-output nullability in ASAPOp schemas); a key-only HAVING that every consumer applies becomes a Filter below the state (SummaryAgg.filter would keep the dropped groups); a value HAVING stays a Filter above the output. avg divides its finalized sum and count accumulators, and the measures of a multi-measure aggregate are realized one by one and joined back on the group keys. ScalarExpr::map_columns renumbers a predicate's columns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol
added a commit
that referenced
this pull request
Oct 13, 2026
… targets Composition substitutes every #561 Pass 1 choice. A measure filter becomes SummaryAgg.filter (the realization rule #608 carried, with its filtered-output nullability in ASAPOp schemas); a key-only HAVING that every consumer applies becomes a Filter below the state (SummaryAgg.filter would keep the dropped groups); a value HAVING stays a Filter above the output. avg divides its finalized sum and count accumulators, and the measures of a multi-measure aggregate are substituted one by one and joined back on the group keys. ScalarExpr::map_columns renumbers a predicate's columns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
Merged
3 tasks
zzylol
added a commit
that referenced
this pull request
Oct 13, 2026
…mputation candidates (#575) * feat(planner): compose a whole-workload candidate from Pass 1 choices Replays #575's composition on the #561 Pass 1 API: choice[i] indexes targets[i].choices.candidates, several targets (measures) of one aggregate are grouped, an exact accumulator becomes SummaryAgg → FinalizeExactAccumulator and a summary SummaryAgg → SummaryEstimate. Pass 1 summaries are unsized, so composition sizes each for its measure's accuracy requirement. Summary coverage is derived from the subtree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * fix(ir): name a finalized exact accumulator after the aggregate it realizes FinalizeExactAccumulator kept the state column's name (`state`), so a consumer reading the sample value (e.g. sum by (job) over a finalized per-series rate) found no `value` column at runtime. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(planner): rank Top-K summary items by the PromQL series identity column Rows that carry the full series identity are closed; Pass 1 rejected them and the runtime needs explicit item columns for keyed summaries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(devtools): stage_pipeline writes real frontend and logical ASAP DAG documents Replays #575's devtools commits on the #561 Pass 1 API: the frontends section is the frontends' workload DAG and logical_asap_optimization holds one composed whole-workload candidate per choice of a per-computation candidate for every target, as flat DAGs, capped by --max-candidates. Example 1 has three candidates (one exact option per target), so the cap test cuts at two. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * feat(planner): compose measure filters, HAVING, avg and multi-measure targets Composition substitutes every #561 Pass 1 choice. A measure filter becomes SummaryAgg.filter (the realization rule #608 carried, with its filtered-output nullability in ASAPOp schemas); a key-only HAVING that every consumer applies becomes a Filter below the state (SummaryAgg.filter would keep the dropped groups); a value HAVING stays a Filter above the output. avg divides its finalized sum and count accumulators, and the measures of a multi-measure aggregate are substituted one by one and joined back on the group keys. ScalarExpr::map_columns renumbers a predicate's columns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * feat(planner): give composed logical ASAP candidates theoretical guarantees Every node of a composed candidate gets a guarantee: exact over exact inputs, a summary estimate's sized family bound over exact input, and None for anything computed from an approximate value, including a value HAVING or any other predicate at any depth (#669 defines composition; plan selection's accuracy model decides). The Example 1 fixture shows them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * fix(planner): keep summary candidates for combinations of approximate values A comparison of approximate operands (quantile(0.99, …) > quantile(0.5, …)) now gets the same BinaryOp candidate over summary operands as arithmetic does, so no combination loses its summary candidates. Automatic selection still needs evidence about the candidate's output: a root guarantee that meets the target, which composition (#669) supplies, or the deployment's empirical output-level accuracy model (plan selection, #576). Certified inputs are not enough, since a difference or ratio of them can be unbounded, so such candidates are kept but not selected under the built-in models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * docs(planner): composition of Pass 1 candidates in the module doc The composition of per-computation candidates into whole-workload candidates is Algorithm 1's SUBSTITUTE; its helpers are named so. The module doc describes it and states that summaries are sized here per measure because Pass 2 sharing is not implemented yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * fix(planner): no frequency bound on Top-K or signed Count-Min estimates A heap sketch's bound is per item frequency, not Top-K membership, and a Count-Min bound assumes non-negative weights. A composed Top-K estimate, and a Count-Min estimate over weights not proven non-negative, now get no guarantee; plan selection's accuracy model decides them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * fix(planner): rank Top-K over a closed PromQL child by its labels topk(3, sum by (job) (rate(m[5m]))) listed heap candidates that composition could not substitute: the closed child [job, value] has no series identity. Its item is now the tuple of labels outside the partition, as in the legacy current-series and counter rules. stage_pipeline records a candidate it cannot compose with its error instead of aborting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * fix(planner): substituted outputs keep the aggregate's column names An estimate or accumulator names its value after the operation (quantile_0_99 became quantile, avg sum, a per-series count value count). A single-measure substitution is now renamed to the aggregate's columns, as join_measures already did; a renaming projection keeps its child's guarantee. The logical_asap_optimization test compares column names too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * fix(planner): substitute subqueries inside measure filters and HAVING A subquery in a substituted aggregate's measure filter or own HAVING now reads its rewritten plan, and a summary state's guarantee requires every input, its filter's subqueries included, to be exact. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * fix(ir): a filtered quantile aggregate is nullable As its summary estimate is: no row passing the filter gives NULL. The filtered quantile queries join the SQL candidate composition test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * fix(planner): compose SQL avg as a keyed join of sum and count A PromQL BinaryOp matches Utf8 labels: it merges NULL and '' groups and rejects other key types. For SQL rows the finalized sum and count are joined on the group keys (NULL keys match) and divided as SQL, NULL where the count is 0 (a filtered-empty group, an empty global input). PromQL keeps the BinaryOp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * fix(planner): count states read a column non-null on every counted row The count state behind avg-as-sum/count and SQL count read the sample value, which for SQL rows is `value` or the only numeric column: avg(c) divided by count(value), count(*) counted a nullable column, and with several numeric columns the composed DAG was invalid. The runtime skips NULL updates, so: - avg's count reads the averaged column (non-null, or the candidate is pruned); - SQL count reads a column its filter proves non-null (count(col)), else a non-nullable column; with none it is Unsupported; - an unresolvable state input is Unsupported, and a composed DAG is validated before it is returned. PromQL counts still read the sample value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test(planner): a subquery in an aggregate's own HAVING is substituted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack: Wave 3 chain: #605 → #607 → #608 → #609 → #610
Rebased on main d4869a7 (DF 54).
Stack: … → #605 → #607 → this PR
Why
Part of #509 and #580 (J, filtered aggregates); Q47 (a). Pass 1 listed alternatives for a filtered single-measure aggregate, but
realizerejected every non-pass-through choice ("filtered or HAVING aggregate"). So these alternatives failed in Stage 1, and the query could only run as written, which #607 makes possible. This covers SQLFILTER (WHERE …)andCOUNT(y)over a nullabley, which is lowered as a count filtered byy IS NOT NULL.What
SELECT g, COUNT(*) FILTER (WHERE x > 0) AS c FROM events GROUP BY gat ε = 0.1, δ = 0.01:Countacc, CMS, CountSketch, UnivMon and HydraCms fail:unsupported local realization: filtered or HAVING aggregateSummaryAggcarriesfilter: x > 0. An alternative binds in the executor exactly when its unfiltered counterpart does. TheCountaccumulator and HydraCms execute to a 2, b 0, c 1, matching the exact plan.For a filtered
SUMand a filteredapprox_percentile_cont, the exactSumaccumulator, KLL and DDSketch execute to the exact answer. They read NULL for the group with no matching row.Selection: the Examples 1, 3a, 3b and 4a stage documents are byte-identical to #607's, and the committed Example 1 fixture is unchanged. Example 2 has no filtered measure (
planner_layering_example2asserts this), and its tests are unchanged. I did not tune costs.How
Pass 1 (
logical_candidates::realize): a single measure's filter becomesSummaryAgg.filter, which reads the same input rows. This applies to every family, Hydra, and tumbling panes, since each pane carries the filter. HAVING stays rejected. The SQL frontend lowers it to aFilterabove the aggregate anyway. A whole-expression (absorbing) target is never filtered.Coverage stays whole-source (Check declared summary coverage population against subtree filters #570: the declared population is trusted). A filtered state holds a subset of its source, so whole-source over-states its population. That is the conservative direction:
A
column = literalfilter could be declared as that population, but nothing reads populations from Pass 1 yet. I did not do this.Types:
FinalizeExactAccumulatorand a quantileSummaryEstimateover a filtered build are declared nullable, like the filtered aggregate each realizes. Without this, a filtered exact SUM read 0.0 for an empty group.Tests
These failed or did not exist on #607:
filtered_aggregates::pass1_offers_filtered_count_alternatives: the six alternatives, the filter on each, bind parity with the unfiltered plan, and execution matching the exact plan.filtered_aggregates::pass1_filtered_sum_and_quantile_alternatives_execute: SUM (exact acc) and percentile (KLL, DDSketch within 2%), with NULL for the empty group.stage_pipeline_selection::sql_filtered_aggregates_dp_equals_exhaustive: a filtered grouped count × a filtered percentile, 18 combinations. The DP equals exhaustive, and none is rejected while being built.summary_coverage_e2e::sql_worked_example: the docs: ASAP primitive schema and summary coverage (definition + selection) #573 §4.2.2 worked example,FILTER (WHERE latency < 100)over a renamed, filtered subquery. Its KLL state's selection islatency ∈ (−∞, 100),r ∈ {us}, and the definition keeps no filter on theSummaryAgg. The test reads the state from the Stage 1 candidates, since the selected plan for such a small query may build no summary.filtered_aggregates::exact_filtered_sum_is_null_for_groups_without_matchesfrom #607 now selects the filteredSumaccumulator, and it still reads NULL.Not in this PR
Aggregatestays rejected. The frontends never emit it; SQL HAVING is a post-aggregateFilter.COUNT(*)does not bind (keyed summary weight must be a finalized value column), and filtered ingestion-time precompute is unsupported (executor: filtered aggregates and summary builds #607).Gates
cargo fmt --all --check,cargo clippy --workspace --all-targets --all-features -- -D warnings: pass.cargo test --workspace: 1,652 passed, 23 ignored (feat(planner): Pass 1 offers HydraCms for grouped approximate counts #605: 1,642; executor: filtered aggregates and summary builds #607: 1,649; +3).tools/dag-viewer): 29 ran, 6 skipped.Refs #509, #580, #605.
🤖 Generated with Claude Code